“Does Charter School Attendance Improve Test Scores?”: Comments and Reactions on the Arizona Achievement Study

In a recent report, Solmon, Paark, and Garcia (2001) seek to identify the impact of attending charter schools on student achievement using data from Arizona. Based on a sophisticated statistical analysis, these authors report that charter school attendance increases test score gains of students. This note raises some questions about the interpretation of the results reported and some questions about the empirical approach and underlying data. First, the report relies on a 2-x-2 evaluation design with type of school (charter or traditional) attended in a base year as the rows and type of school in the ensuing year as the columns. The report compares the observations in a cell of the design matrix to all other cells. This note questions the validity of that approach and suggests that the way that the data were constructed allows comparisons only across the rows. Second, the note questions whether grade level was used in the data matching procedure used to construct the comparison sample. Third, the note questions whether sex was used as a covariate in the outcomes equation and whether building or district fixed effects were used to control for unobservable factors at those aggregate levels. Finally, the note suggests that marginal costs are more appropriate for a cost-benefit or cost effectiveness analysis than average costs, which were used in the summary section of the report.